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(54) TttlK A NEW EDOB-DBrnmON BASED NOISE REMOVAL ALGdUTHM 
(57) Abstract 

What is disclosed is a method for lemoving noise by 
distinguishing between edge and non-edge pixels and applying a 
first noise removal technique to pixels classiiicd as non-edge pixels 
and a second noise rsinoval tedmique to pixels classified as edge 
pixels. The methoddogy operates on images while in a Color 
Filter Anay (CFAO domain priw to color interpolation, and uses 
techniques suited to flie classification, whether edge or non-edge. 
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A NEW EDGE-DETECnON BASED NOISE REMOVAL ALGORITHM 

BACKGRODND OF THE imnriVTf f>TVf 

1. Field of the Inventinn 

The invention relates generally to the field of ima^ processing. More 
specifically, ttie inventioii relates loocesaaig of images ^xteiated by a di^ilal imaging 
device. 

2. Description of the Related Art 

In fee at of image processing, raiw images of an object/scene captuied fiom a 
sensing or capture device are often subject to varying types of "noise" (dements not 
present in the object or environment which may nonetheless appear in the image). The 
noise present in an image may be due to ttie characteristics of the imaging system soch 
as the sensor or processmg stqs subsequent to the initial image capture which may add 
noise viiulettying to SK^eve a difierent purpose. The propaties and chaiactorisdcs 
that would indicate that a pixel or regitm of inxels is "noisy" and the pr<q)erties that 
would indicate a pucel or re^on of pfocek is an edge or a fine detail of ti» image are 
difficult to distinguish. Thus, a fimdamental problem wifli the removal of noise is that 
oSm a removd of is indicated as ndse niay actually be a removal of fine edge or 
detail. Iffte fine detail or edge is renM)ved, a Uuiring effect may occur witfam that 
region of fliB image. Furtbor, in color images^ Oe blurring effect leads to a bleeding of 
one color across tteeci^ to atioflieriMxeKs). N(Hset»noval procedures that were based 
upon linear filt«»cing techniques suffered greatly fiorn tiiis malady and thus, a class of 
filtering techniques based on ranked order statistics such as tbe median were developed. 

The median filtear ranks m order the intensitj' values beloi^i^ to a pfacel P (for 
winch the filter is being aiq>lied) and pucels in a particular ndgbborhood or aloug a 
. particular vecttv about a pixel P. For example, a median filter (applied in a particular 
direction(s) through Hin pixel to ndghbcning pixels) q>plied to sample values induding 
and about fte pixel P of {12, 13, 200, 50, 14} would first be ranked in order as {12, 13, 
14, 118, 200}. The so-called uni-diiectional FIR median l^brid fiher would rephice the 
original pixel location P that had a value of 200 witb die median of tt» sang)le set whidh 
is 14. Thus, the output vector, after the filter, would be: {12, 13, 14, 50, 14}. If the 
vahie 200 weie in fact part of an edge radier than noise, the anootiung caused by 
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applying the fihw as shown m flie output vector values would decimate the edge 
feature. 

Sevetal iinpioved median filt^ have developed to compensate for this 
problem. One particular such median filter, toe multilevel FIR (Finite Impulse 
Response) median hybrid fitter repeatedly takes the median filter in each direction 
about an image and applies at eadi filter the ari^nal input pixel. The multi-level median 
byrbrid filter has avraa^g sub-filters that reduce the burden of sorting operations by 
averaging pixels in a particular filter direction, and then performing the median 
computation upon a smaller set of values, such as ttuee. Thus, in a median hybrid 
filter, two neighboring west pixels would be averaged and the result fed to a median 
filter along with the average of two neighfaormg east pixels. The third infnit to the 
median filter is the pixel under consideration for noise removal. In other directions, a 
similar procedure is applied. In a toee-level median hybrid filter, the first level pairs all 
such averted neighboring pixels with vectors m opposmg directions (north with south, 
etc.) and for each pair of direction averages (8 of them) feeds these into a median filter 
also along with the pixel of concern as a Oiird input. The resultuig median values of the 
first filter are again paired and along with the foxel of concern are input to a median 
fiilter. While median hybrid has been shown to wo* quite well in discriminatii^ some 
edges, it is deficient in several respects with regard to edge detection. The median 
hybrid filter does not consider the noianess of the edge hself. In other words, an edge's 
direction, even though eight are employed, cannot be determmed with exacting accuracy. 
For instance, an edge feature may lie at a 33 degree vector from a particular pixel, and 
thus the eight directions are inadequate m detetmining.the edge feature. In other words, 
a smgle pixel may contain a portion that is edge and a portion that is non-edge m the 
non-discrete world ttat cannot be represented in the discrete world of digital ima^ 
When applied to digital images, (he median hybrid filter, if applied eveiywhete to all 
pixels, may propagate noise or shift it from pixel to pixel while attempting to remove it 
since there is noise along the edge feature due to &e noa-canSnal direction of die «^e. 
A curved edge is a petfect example of such a problem. 

When an object/scene is imaged by a setisiag or imaging device such as a digital 
camera, the resultant hnage in captured into a CPA (Color Filter Array) bearing a 
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particular color channel pattern. One oft-used pattern for capturing images is loiown as 
the Bayer pattern, which has color channels as follows, 

GRGRGR... 

BGBGBG... 

GRGRGR... 

with rows thereafter repeating Hbe pattern. 

Thus, in a B^er pattern CFA, each pixel location has an intensity value 
associated only with one of the three color planes (Red, Gretai and Blue) which 
combine to make a full color. The process of estimating the two mijyj j pg color 
cranponents for each pixel location is known in the art as cdor inteipoIatiQn. The ■ 
iatBrpolati<ni of colas often precedes the lemoval of ncHse in color images due to tiie &ct 
1hat most traditional noise reduction or removal tecboiques iare designed to operate 
upon images with Ml color pixel infbnnaiion. The process of color inl^polation itself 
will introduce noise, and in the case of a lUgital camera, where the color intrapolation is 
nK>st likely perforno^ after un2^e compression ai^ download to a data prooessli^ 
system such as a computer, the intramediaiy steps of color resolution coinpanding, 
CQinptesdon (quaatizatioii and encoditi^) ai^ deconipression may add additi<»ial noise 
such that the <Miginal cqXured inrage noise may be blended vnth other noise to perhaps 
lose the distuicfian of being noise and gain the distinctton of being an image feature. 
Perftnming noise renuival on ftie fiiU color pixels attained by fbs color interpolation 
process increases the mstasxy mi processing aeeis of the noise removal process by a 
fector of 3 (since each pixel 1^ thrice the resolution), and thus is diffoult and expensive 
to implement in hardware. Other noise removal techniques attempt to reduce this 
burden by performmg color space conversion a£b& color interpolation into, for instance, 
the YUV space, v/beK <mly the Y (duominance) component is considered &x noise 
removal. However, this too may propagate additional noise b^ond that propt^ted by 
color inteipdlation and ako caniKn be easily be inplemooted in hardvvare. 

Thus, there is a need ft>r a noise reduction fiamewotfc that will not only 
distin^sh edge pbeAs from non-edge pixels, but also one that can woik directly in the 
CFA image domam prior to any color interpolation. 
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SUMMARY OF THE INVRNTTflTH 
A method is disclosed having the stqjs of classifying pixels of a captured image 
while in its Color Filter Array form as either edge pixels or non-edge pixels. The 
method next performs noise removal by applying a first noise removal technique to 
those pixels classified as non-edge pixels and applying a seconii noise removal technique 
to those classified as edge pixels. Bofii noise roDioval techniques arc applied v^e Hie 
image is still in the Color Filter Array form. 

BRIEF DESCRIPTION Off THir DRAWINGS 

The objects, features and advantages of the method and apparatus for tiie 
present invention will be apparent from the following deswip&n in which: 

Figure 1 is a flow diagram on one embodiment of the invention. 

Figure 2(a) illustrates a ma^ for obtaining a fo-st diiection rfiffeiw^tiiii. 

Figure 2(b) illustrates a mask for 6btalnii]@.a first duection differential, 

Figure 3 illustrates the application of he noise removal technique oidepen^iidy 
in each color plane accordmg to (Hie embodiment of the invention 

Figure 4 illustrates the principles of at least one onbocUment of the inventicaii 
as applied to an exemplary image region. 

Vigare 5 is a block diagram of a three-level median hybrid filter as anphyed in 
the various embo^ments of tiie invention! 

Figure 6 is a block diagram of a three-level median hybrid filter as eii5>loyed in 
the various embodiments of the invention. 

E%are 7 is a system diagram of one mbodimc»it of the mvention. 

DETAILED DFJinRTPTf ON OF THF. INVKNTfON 

Referring to the f^ures, exenqdaiy embodimeads of the mventkm will now be 
described. The ^cemplary embodiments are provided to iUustrate aspects of die 
. invention and should not be construed as limiting the scope of the invention. The 
exemplary embodiments are i»imarily desoibed with referotce to block diagraniis ot 
flowcharts. As to the flowcharts, each block within the flowcharts represmts both a 
method step and an ^)paratus elemmt for perming ibe msOxod stq>. Depraiding 
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upon the inq)lementation, the conesponding apparatus element may be configured in 
hardware, software, firmware or combiijations thereof. 

When an image captured by a sensor or camera or other imaging device is 
analyzed, it can be said to have portions of the image which are "noise" or "noisy" 
while other portions are not. The detemiination of what constitutes noise is difficult 
and not an exaction process. Noise may be confused with a detail such as thin line or 
"edge" portion of the image. The removal of noise in a captured image is optimal when 
detail iiifonnation is stiU preserved whOe noise is rmoved. In traditional noise lemoval, 
eexh pixel, whether having edge features or not» is treated by the san^ noise removal 
technique due in large part to this relative inability to distingoish noise fym «d^. 

In color imag^ such as ftose captuced'into a Bayer pattern CPA (Color Filter 
Ain^X ^ step of color interpolation may additionally introduce noise and tiuis, Hat 
original noise due to tbte imaging device itsdf is far&XBi perturbed. The addition of 
various types of noise, at eacli st^e of the imagjije process fiom cqmiie to rende^ 
. presents additional difBcuBy since the noise pattegns are cnmhirted in an i m l g w y iiyn way, 
Ilie resulft^ image wiU have dements &at cannot be distinguished as 
when the B^er pattern CPA is sidgected to color inteipolation, as is required a priori 
for most noise rraooval techniques* &e colors will bleed fiK»n ed^ to ed^ and feature to 
feature. For instance, consider a Uue edge feiOuze lying on a yellow bad^und. Due 
to noise a red pixel m^ lie somewhere on the edge boundaiy. Due to typical color 
interpolation methods, ^e blue and yellow will be avocaged togefliar with ^ red to 
fimn a fiill i»xel with blue, yellow and red sranewteie in tte ndgbboiiiood (£»nce red 
ap^tears to exist as a feature, albdt sinall, withk the image region), and thus, a false 
cdor pixel will be generated whidi may not tfaereafio: be disconible as noise to a noise 
removal process. - 

Thus, it would be desirable to devise a noise renurv^ technique that works m 
the GFA dtnnam i»ior to the process of color interpolation so that the additive noise 
and color bleeding problem can be drcumveaoted. Ftirlher, tte technique should treat 
and cotiader jMxels belonguig to e«^ features in the hna^ diffei^ 
belongbg to «^ features. Also, the teduuquB(s) used should be c^qiable of fest 
l»rdware in:q>l«nentafion such that it can be implemented, if desired, in a distal camoca 
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•wheio the CFA is generated. One such possible noise removal methodology is shown 
in Figure 1 and in various other embodiments discussed belov/. 

Figure 1 is a flow diagram on one embodiment of the invention. 

First, an initial localization region within the captured image is defined (step 
1 10). The initial locaIi2ation region will be a M by N rectangular re^on of the cj^>tured 
image where M and N may be equal if desired thereby dejBning a square region. The 
localization region as a percentage of the total c^tured image size may be large or small 
depeiiding upon the desued implementation and available hardware/software resources 
and speed constraints for how fast the image is desire to be processed for the removal 
of noise. Where the unage is more uniform in characteristics over a larger region, the 
. localization region may be defined large. However, if fine detail disodmination is orticial 
or the image shows wide variations in ciiaiacteristtics within small legtons, 
localization region .can be defined analler. 

Once the localization region is defmed, the next step is ta determine the gradient 
value associated with each and every pixel in the localization region by applying some 
mask or operator (step 1 20). This mask or gradient operator is applied to a small 
neighborhood about each pixel, ordinarily whh that ndgfabothood as well being of 
smaller size than the localization xe^n. Figures 2(a)-2(b) and associated description 
iUustrat© tiie parameters and configuration of swA a mask witii a 3 by 3 neigjiborhood 
about and iocluding the pixel whose gradient is being det^mined. By repeatedly 
applying tiie operator or mask to each pixel in ttie localization region and determining 
the gradient thereby, gradient information for tiie localization can be cotnpleted. 
In most ^adient-based edge detection, Ate gradient value is nomialized over die oitiie 
image by dividing the gcadifflot by tiie maxmnim gradioit in the entire image. However, 
such a normalization would lequiie, if in^lemoated in hardware, a two-pass approaiHx 
yAunx giadieiit information for all pbcsls va the ea&K captured image must be 
determined and tben, each gradient divided the masmum gradient vahie fou^ 
the image. In tiie various ooabodimentsofthe invention, the gradient nuiyin^^ 
ncmjialized by tiie maximum witiun tiie localization region. 

Once tiie gradient mfbnnatioQ is determined, a tineshold shouM 
(step 130). The tihreshoid value may be selected as a single valte applicable throughout 
the image or as varying from locaUzatioiir^on to loGsdizati(m region. Furtiia^,the 
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threshold may be preset to a particular value for every image captured by the imaging 
device. Such a preset threshold is not very accurate and can upset the edge detection 
process, but according to the current state of the art, is most desirable for ahardware 
nnplementation of noise removal. The selection of this threshold is not a subject of this 
invention, but if the gradient information of every pixel within fee localization region is 
first determined it may be possible to dynamically select a tfai^told value. 

Whatever the selected threshold value,' the gradient (or normalized gradient) is 
compared against that threshold value (step 140). If the gradient (or nonnalized 
gradient) racceeds the thr^hold value, the corresponding pixel can be dassilBed as an 
"edge" pixel wUdb is a {sxel that belongs to a edge feature of fee unage such as a line . If 
not, the i»xel is ctas»fied as a non-edge pixel. According to one embodiment of fee 
invention, fee edge pixel and non-edge pixel each recdves a dlffiereiit noise 1^ 
treatment. While one mefeodofedge detection, nao^ gradient based ec^ detection 
has been presented in fee flow diagram of Fisnre 1, any suitable edge detector inay be 
utilized. For instance, fee wdl-known Canny edge detector, which uses Gaussian masks 
convolved wife fee hnage space in clas^fying edge and non-edge inxels, may be used in 
conjunction wife the technique of F%iire 1. and if cipible of implemenlation in fee 
desired applicaticm, will pnmde good edge disoinmHlion. For a distal inu^ng device 
sudb a portable stiU digitd camoa, an edge detector such as (te one pies^ted in 
Figures 1, 2(a) and 2(b) are more qptimal due to fee ^ed of thdr computation and 
use of less dlicon area within the device. 

By classifying pixels as dfeer edge pixels or non-ec^e pixels, they may, 
according to one embodiment of the invcaition, be feen subjected to dUQfeient noise 
removal tedunque and treatment Hius, aocordmg to step 160, a noise removal 
technique sudti as multi-level median hybrid filtering may be ajqdied upon the pbcd (see 
Figure 5 and associated description) iffedpixd is a non-edge pixd. Thistreatment 
avoids the problem posed 1^ median filtering of bhuring and dechnatmg edge features, 
suice the pixel does not bdong to an edge feature. The median filtering, when applied to 
the non'«dge pixel will have the desired effect of mnovmg noise without blurring the 
edge. 

Since noise may also be present in a pbcd feat is classified as an edge pixd, fee 
edge pbcel is subjected to a smoofeing tedunque sudi as linearly averagnig nd^boring 
7 
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pixels having a similar iatensity value (step 1 SO). Ibis smoothing ensures that an edge 
is not completely decimated and in feet, may enhance the edge in appearance (i.e., 
sharpen the edge by smoothing together tfie color (intensity) more uniformly). The 
linear averagmg technique replaces the pixel under consideration widi a linear average of 
itself and neighboring pixels that have a similar intensity. One method of discovering 
which pixels qualify to be of "similar intensity" is described below with respect to 
Figure 6 and its associated description. Essentially, the goal is that two or more edge 
features that border each other sliould not be confused when averaging so ^ edge 
features are not combined together improperly. 

Next, if any pixels are remaining to be treated to noise removal within the 
localization region (checked at step 170), the next pixel's information is fetched (stqj 
175) and steps 140-160 are repeated. Once all pixels within the localization region are 
treated to noise removal, a new localization r^on may be defined (step 1 SO). 
Processing then continues with steps 120-170 repeating for die localization tibreshold. 
Preferably and for a &st implem^tation, the localization regions should be exciuaive, 
but may overlap if desired. The noise rsnoval technique described in Fignre 1 may be 
applied directly in the CPA domain to pixels in Bayer pattern prior to any color 
mteipolatioa. Thus, the Figure 1 technique, if so employed, will reduce the amount of 
propagated noise throughout the various stages of processing the image. 

Figure 2(a) illustrates a mask for obtaining a fnst direction differential. 

One means of determining the gradient for a pixel is to i^>ply a mask A wfaicb 
essentially fmds the "strength" of Hie pixel about a neighborhood boundary that the mask 
covers. This strength is essentially a dif^reatial in a particular direction along two 
boundaries of a pixel. This Gist direction, vibsa combined with a similar computation for 
the strength along a second direcUon boundary gives a good estimation of the relative 
strength oftbe center pbcel as {^gainst its ndgfabors. IiiFi^re2(a),3by3ma^is 
illustrated that has nourzero coefiBcients in its 1^ and 3"^ columns, and zero coefBdmts 
in the middle am 2"^ column. The mask A is shown to have six non-zero coefiSciente Al, 
A2, A3, A4, A5 and A6. When the mask is applied to an Image region I having pixels 
with intaasity vahjes 10, II, 12, 13, 14, 15, 16, 17 and 18, the resultant differential Ax is 
expressed as: Ax=Al*ia+A2*I3+A3*I6+A4*I2+A5*I5+A6n8. The differential Ax can 
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be used in calculating the gradient of the pixel -with mtensity value 14, shovra to be in the 
center of the image region I. Ordinarily, ti» A will have coeflBdoits that summed 
together give zero. In the well-known Sobei made, the coefScients Al, A2 and A3 all 
have values of -1 while the coefficients A4, A5 and A6 all have vahies of +1 , Thus witti a 
Sobel mask's coefBcients, Ax, the differential, is Axpn+IS-flS-IO-D-Id. Thus,-1fae choice 
of mask coefficients, which is known m the ait to achieve a meaniTt gfiil gradient value, 
sinq>Hfies tiie differential con^utatiMi to a mrae addition and subtraction of pixel 
inteiudty values vdddi can be inqdernentedeaidly in faaidvt^ Thus,tt» 
differential' becomes easy to imfdement and &sL A different mask, such as the Prewitt 
ma^ if {Qiplied can be as easily implonented The Prewitt mask has coefBcients of Al 
andA3equalto-landanA2coe£Bcientof-2. To minor, the Piewitt mask has 
coefBcients on the odier directional boundary of A4 and A6 equal to +1 and a coeSlcient 
A5 of +2. Using a Prewitt mask, the Ax or differential vahie becomes Ax!=I2+2*I5+I8-I0- 
2*13-16. Tliough a multiplicatioii by 2 is introduced, this can be implemented eaaly by a 
shift operation of IS and 13 or t)y splitting llie computation Ax as Ax"I2+IS+IS+I8-I0-I3<- 
13-16 if only add and subtract operations are implemented in tiie hardware. 

The gcadi^ operation is used m edge detection abce it is assumed tiiat an edge 
has a particular direction witbintiieunage^pace. The^xels along tiiedde boundary 
(one dde-IO, 13, 16 and another side-12, IS, 18} of the piswl should have a strraqgOi or 
relative combined intend level less than the jixsl 14 itself if 14 is to be ocmsddered an 
edgei»xel. In this regard, a fast dhectiondifferentia^essentiaUy in the hoii^ 
direction (the mask has values determining tiie strength on the left and tigfatsides of the 
unage) may be computed as given above. Since the hnage space is m two dimmaions. 
vertical and horizontal, onfy one more direction differentud about the phcel 14 need be 
computed. 

F^iire 2(b) illustrates a mask for obtainmg a second dhection difiEbroitial 
accordhig to one embodunent of the invrastion. 

In F^ure 2(b), a second 3 by .3 mask is illustrated tiiat has non-zero 

coefficients in its 1^ and 3"^ rows, and zero coefficients in the middle or 2"** row. The 
mask is diovra to have six non-zero coefficients Bl, B2, B33, B4, B5 and B6. When 
tiie mask is applied to an Image region I having pixels witii intensity values 10, 1 1, 12, 
13, 14, IS, 16, 17 and 18, the resultant differential Ay is e^qnessed as: 
9 
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Ay=Bl*I0+B2*Il+B3*I2+B4*I6+B5*I7+B6*I8. The diflFerential Ay can be used in 
calculating the gradient of the pixel with intensity value 14, shown to be in the center of 
the image region I. Ordinarily, the maskB will have coefficients that sunuried together 
give zero. In the well-known Sobel mask, the coefficients BI, B2 and B3 all have values 
of -1 while the coefficients B4, B5 and B6 all have values of +1. Thus wife a Sobel 
mask's cpeflTicients, Ay, the differential, is Ay=I6+I7+I8-I0-Il-I2. Thus, the choice of 
mask coeflScients, which is known in the art to achieve a meaningful gradient value, 
simplifies the differential computation to a mere addition and subtraction of pixel 
intensity values which can be impieraentBd eaafy in hardware by adders. Thus, the 
differential becomes easy to implement and fest, A different mask, such as the Prewitt 
mask, if s^plied can be as easily implemented. The Prewitt mask has coefficients of Bl 
and B3 equal to -1 and an B2 coefficient of -2. To mirror, the Prewitt mask has 
GOefBcients on -die other directional boundary of B4 and B6 equal to +1 and a ■ 
coefficient B5 of +2. Using a Prewitt masdc, the Ay or diff^ential value in the seccmd 
dh'ection along the image boundary beconiesAy=I6+2*I7+I8-I0-2*Il*I2. Though a 
multiplication by 2 is introduced, tihis can be implemented easily Ijy a shift operation of 
15 and 13 or by splitting flie computation Ax as Ay=I6+I7+I7+I8-I0-Il-Il-I2 if only add 
and subtract operations are implemented in Hie hard\rare. Preferably, the mask A used 
to jSnd the Ax or &st direction differential ^uld be llie same type of maskfi that is 
used to find the second direction differential. It may be de^rable to tse a Sobel mask in 
one direction and Piewitt in auo^ to c(mq)ensato since 
yield square pixels, but rattier, rectangular pix^ or pixels of other isdu^ In such a 
case, the geomeby of a pbcel nu^ be accounted for in ttie dioice of mask ooeffidents. 

With flie second direction difra:ential Ay obtained as &v&a aibove, both 
directional bouodaiy pahs horizontal and vertical are accounted fbr. Then the smdi^ 
of 14, the central pixel in flis image space I is the noagnitiide of &e vect^ 
vector \^ch is the sum ofthe vectors obtained in each directian. Hence, flie gradient G 
isshnply 'y/^c^T^, or the inag^iitude of that vector fi^ed by comt^ 
Ay. The gradient of each pixel mvy thus be obtained by considering a neighborhood, 
preferably a small 3 by 3 neighborhood Hbat surroimds each pixel, and then by applying 
a mask upon it in each of two directions and their corresponding boundary pairs 
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essentially thereby including every immediately adjacent pixel in the gradient 
detennination. As mentioned above, the gradient vahie may be nonnalized for the 
maximum gradient present in the im^e so fliat relative weight is given to the gradient 
measure. For instance, in a captured image of a scene with more texture rather lhan hard 
^ges such as lines, the gradient vali^ may lie wi)hin a narrow n^on. If these ai« 
nonnalized, to the maximum gradient, fbsa Ifas gradient values obtained can be more 
effectively analyzed. 

Figure 3 illustrates tiie plication of the noise removal techni(pies to each 
color plane independently. 

Since the noise removal technique presoated in various embodiments of the 
invention operate directly upon ttie CFA version of tiic captured image which is 
essentially the raw captured image, it may be deared in one embodiment that Ihe color 
planes may be separately considered. Hius, considmng the 6 by 6 pixel Bay» pattern 
region of the image, the Blue (designated by "B" in Figure 3) color plane (i.e., pixels 
assodated with Ihe color blue) may be separated out fiom ifae CPA and have the noise 
i«aa.oval tec^que desoribed above ai^lied to it sq>aiate <^cotcv Thus, the 
pixels assodated with the blue plane at locations (2,1), (4,1), x(6,l), (2,3), (43), (63) 
and (2,5), (4,5); (6,-5) c»n be arranged so that fh^ are each adjacent one anc^. 
Likewse, other neighboring pixels in the Blue plane (not diown) of Uie CFA may be 
combmed with the nine pixels illusfiated to form a Blue sub-image. The' noise removal 
fianx^rk described m the various embodiments of the invention may th«i be applied 
to tins Blue sub-unage directly and ii^ependentily or o&nst color planes. 

In the Bayer pattern, Gieen i»xels havQ roughly twice tiie rqnesentation of 
»1ii«-.Blue QiJRsd pixels in tiie CFA. Often, it is usdiil to split the green pixels into 
tho% ii»t accompany a row of altianoting Blue pixdls (a first Gieen labeled 01) and 
tiiose that accompany a row of altnmaling Red pixels (a second Green labeled 02). 
Thus two Grera color planes, Gl and 02 may be extracted from tiie CPA. Thisis 
useful since the number of computations for each color i^ane will then be identical, If 
Gl and 02 planes were considered together when {q)p]ying the noise removal technique 
thereon, there would need to he a row and column oi&et to combiin Ifae 01 and 02 
pixels (since lh^ are on (fifTerent rows and cohmms), and fuitb^ fte combmed sub- 
hnage of a unified 01 and 02 would be twice as large m pixel count as the Blue and Red 
11 
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sub-images/leading to the need for a different, larger sized memory. For the purposes 
of analyzmg the image, it is useM to consider the two Green color planes separately 
since the Green is vital in determining the chrominance (color content) pr^nt in 
neighboring pixels. This feet can be of use in techniques such as color interpolation or 
color resolution companding. To preserve the chrominance mformation in a more 
effective way for later use, therefore, it I desirable to consider the Gl and 02 color 
planes separately for the purpose of noise removal. If the noise is removed fiom the 
Gl and G2 sub-images mdependenUy rather than together according to the various 
embodiment of the invention, thai the noise removal will more accurately reflect the 
relationship of details to tiiek original color. The edge features have dkection, and a 
combining of Gl and G2 pixels, which always lie at a 45 degree offset fiom one another, 
would mtroduce a 45 degree shift in the orientation of the edge. 

, Figure 4 iUustrates the principles of at least one embodiment of the invention 
as applied to an exemplary image region. 

Figure 4 shorn a 6 by 6 pixel r^ion of a captured image. A pixel x at a row 
location "i" and column location "j" will be represented hereafter as x(i j). The 
exemplary image region shows both pixels that have already been classified as edge 
pixels (designated by an "E") and ntm-edge pixels (designated by an "NE"). The edge 
pixels are finther designated by subscripts faidicating vMch edge feature the edge pixd 
belongs to. For instance, a first darker edge feature (designated by Ei) may have 
intensity values ui the neighboihood of 200, while a second lighter edge feature 
(designated by Ej) may have intensity whies in the neighborhood of 50. Baah of pixels 
E, and Ezbelongto two different edges of the image. TTie non-edge pixels have not been 
sub-classified into feature, and the classification of edge feature for edge pixels-shown in 
Figure 4 is noerely to ud explanation and is iK>t a lunitation or requirement, of the 
invration in any (anbodiment 

Accdidh^ to one embodimait of the invention, the edge pbcds and nonredge 
pixels vnthin an image are subjected to ^SSkxaat noise mnoval techniques. In one 
embodiment, pbceis that are not classified as belonging to an edge (NE m Figure 4) are 
^lied upon by a first lioise removal technique. In one embodiment of the invention, 
this first noise removal technique is multi-level medianhybrid filtering. The operation 
of such filtering is detailed below with rc^t to Figure 5. All pixels classified as edge 
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pixels are subjected to a second noise removal techaiique diffeient from the first and 
having better edge preservation/wihaacemail diaiacteristics than the fust technique. In 
one embodiment of the invention, this second technique, applied to non-edgfe pixels, is a 
linear averaging of adjacent ^els having similar imensity values as the pixel under 
consideration. 

For instance, refenrihg to Figure 4, the pixel NE(3,3) (non-edge pixel at row 3, 
column 3) would be treated with the m«iian hybrid filtering technique, and is described 
m detail below with respect to Figure 5. The output NE'(3,3) of the median filtering 
process would become the new noise rranoved pfatel at location (3,3). The median 
filtering process wduld utilize all the {nxels, \\iietber th^ are classified as nim-edge 
pbcels or edge pixels, wiftin the ndghbcnrhood (from jrixel locadon (1,1) to (5,5) about 
fbs fHxei NE(3,3) when perfinming its noise removal process. 

By contrast, the edge Ei(3,^ would be api^ed t^ by a Uoear averaging 
of only oth»r edge pixels of sunilar intensify (edge i»xels belonging to tib same edge 
feature) to rranove the noise present in E,(3,2). Of die eight pbcels adjacent to Ei(3,2), 
only three classify as belonging to fte first edge featurc, lamely, pixels E|(2,2), E,(4,2) 
and £i(43)< Thus, the linear avraa^ng noise ranoval technique would average togedier 
four pixels (three adjacoot and the edge pixel heaag i^noved of noise)- Ei(3,2), Ei(2,2), 
Ei(4,2), and E|(4,3). Referring to the second feature, it should be noted tliat the pixel 
E2(6,2) has three adjacent edge pixels, but only one edge pixel £2(6, 1 ) that belongs to 
the same edge feature (i.e., has a similar intensify value). The other two edge pixels 
Ei(5,3) and Ei(6,3)that are adjacent pixel 1^(6,2) belong to a first edge feature, and tibais 
should not be averaged with the pixels of the second edge feature. Otherwise the 
second edge feature would be decimated wiUi its mtensity value contaminated by a non- 
relevant and dissimilar intensity value or values. The methodology shown in Figure 6 
and detailed in the associated description below circumvents this situation by discerning 
edge pixels of similar intensity fiom other edge pixels. Figure 4 shows a thkd edge 
feature (having pucels de^gnated E3) and fourtii edge feature (having pixels designated 
E4) which should be as caiefidl^ treated to avoid contammation fitom ^samilar edge 
pixels. 
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Figure 4 shows only a portion of the image, but with regard to pixels lying on 
the physical edges of the image (first column, first row. last column, last row, etc.) 
these may simply not be processed for noise removal since a portion of the mask(s) 
will have no values, and thus, ineligible gradient information would be attained thereby. 
Altematively, h may be desirable to use the dark current or reference pfacels (of which 
. there are usually several rows and colunms) to substitute for missmg values for edge 
pixels. These dark current pixels are not conadraed part of the final image CFA 
generated by the imaging device but aie nonethel^ captured by the sensing device to 
aid in determining pixel values. 

Figure 5 is a block diagram of a three-level median hytnid fStter as ecapioyed in 
the various embodiments of the invention. 

Accordujg to one anbodiment of tiic invention, a pixel x(ij) if classified as a 
non-edge pixel is applied upon by a noise removal technique that utilizes a multi-level 
. median hybrid filtering process. A toee-level median hybrid filter is designed for a K 
by K block about an input pixel x0 j) that uses Unear averagmg subfolders to obtain the 
various values of the set for Which the median is determined at the first stage or level of 
the filter shown in Figure 5. Thus, if K=4, a 5 by 5 neighborhood about and including 
the pbcel x(ij) is considered for median hybrid filtering. Accordmg to one embodimoat 
of the invention, the median hybrid filter of Figure 5 would be applied to the pixel x(i j) 
only if the pixel x(i j) could be classified as an non-edge pixel, according to gradient or 
mformation. Otherwise, if pixel x(i j) were a edge pixel, the nearest neighboring pixel 
averaging method discussed below with respect to Figure 6 would be applied. 

The averaging subfilters (not pictured) sinq)ly take the aveta^ of a numb^ of 
neighboring pixels as desired by the si2» of the neighborhood. The svm^a^ subfiltm 
are implied to a vector of pixels in each of the ei^t diiecfions (nortb, south, east, w^ 
northeast, southeast, southwest and northwest). Hence, Hao averaging subfiltnrs 'woidd 
produce the following output values YDiitGcnc»i(ij) as foUows: 

YEASi(ij)=(x(ij+l>+xCiJ+2)y2; 

YsoumCij)^ j)+x(i+2 j))/2; 
YnokihO j)= (x(i-y >txa-2j)y2; 
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YsouTHEAST<U)= (xO+l J+l)+x(i+2j+2))/2; 
Y5QtjmwESTCjJ)=.(x(i+lj-l)+x(i+2j-2))/2; 
YNORTHEAST(iJ)= (x(i-l j+l)+x0-2j+2))/2; and 
YNORiHWEST(iJ)=(x(i-lj-l)+x(i-2j-2))/2; 

These outpm values are ibea fed into the median filters along with the center 
pixel value x(i J) vatiousiy as ^own in Figure 5. By using an average sub-fdter, die 
median operation is lednced to a sort of only fluee values at each stage of &e median 
hybrid filtering j[»Dcess. 

Figure 5 diovre that the first level of ifae tihree-level median hybrid filter.has a 
set of median Sltsrs 510, 512, 514 and 516. Median filter 510 has an output Ml that is 
the median of fliree values, namely, YEAsr(ij), YwestOJ) and x(ij). Median filter 512 
has an oulput M2 tiiat is the median of three valiws, namdy, YNORnKij), YsouthO. J) 
and x(ij). Median filter S 14 has an oulput M3 that is the median oftfaree values, 
namely, YNORrHEASi<kj)> YsoumwESrOJ) and x(ij). Fmally, median filter 514 has an . 
output M4 that is the median of three values, namely, YnortowestCU), YsooiHEAsiCy) 
ahdxOj). 

At the next or second level, ttere are a set of two median filters 520 and 522 
wluch take as input, the miedkn filto outputs Ml, M2, M3 md M4 and flffi 
inxel value x(i J). Median filter 520 provides an ou^ M5 that is tfie median of Ml , 
M2 and x(i j). Likewise, median filter 522 provides an ou^ut M6 th^ is the median of 
M3,M4andx(ij). 

At the final or fluid' level, a angle median filter 530 provides the output pixel 
x'Ciji) that is the median of the outputs of the median filters fiom the second level, MS 
and M6, when combined with die original pixel x(i j). Thus, the noise removed pixel 
x'{i j) is equal to the median of MS, M6 and x(io). The noise removed pixel will, in one 
alternate embodiment, be stored in a sejrarate array to not interfere with the remainder 
of the noise removal process for other pixels. In this manner, the original pixel space is 
maintained so that &e noise removal process may be iterated and adjusted to provide 
better results. In another alternative, the noise removed pixel will replace the original 
pixel immediately so lhat the noise removal process is recursive. This has the 
advantage of ^leed over an iterative method but also introduces uncertainly as a 
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misclassified pixel will tend to propagate its mischatacterization to aifect ofher pixels. 
Thus, the on-the-fly compute and replace method is not guaranteed to provide good 
results, though ia some cases it may improve the noise removal. 

Referring badk to the example of Figure 4, the non-edge pixel NE(33) would, 
according to one emhodiment of the invention, be subjected the three-level median 
hybrid filer. As such, Yeast(3,3) = (NE(3,l)+E,(3,2))/2 and Ywest{3,3) = 
(NE(3,4)+E4(3,5))/2. Likewise, the other averaging subfilters can be applied about the 
location (3,3) to obtain the various subfilter ou^uts. The results are then fed into flie 
median filters in the various levels ^ong witfi NE(3,3) at each filter m each of those 
levels. The resultant intensity value NE'(3,3) becomes die noise-removed vrasion of &e 
original captured/decimated pixel NE(3,3)- 

The three-level median hybrid fiher will essentially closely preserve the ori^toal 
non-edge pixel value because the median at each stage or level of the filtering process re- 
introduces the original pixel x(ij) as one input at each of the median filtere. Thoi^ the 
three-level median hybrid filter has been shown to work well with most unages, 
applying it univasally to all pixels of an unage is disadvantageous. The orientation of 
the edge may not be discovraied, as menticH»d above, and further depending vpm the 
signal-to-noise ratio of the image, a noisy edge |nxel be passed through the median 
filter unabated. 

Figure 6 is a flow diagram illustmtmg nearest r^ghbw aveia^ng fiir edge pixels 
according to one embodiment of the invention. 

In one embodiment of the invention, an edge pixel is treated to a different noise 
removal technique than a non-edge pixel. Specifically, an edge pixel needs to retain flie 
values of neighboring pixels that are also edges. When two difieient edge features are 
adjacent and have pixels adjacent each otiier (for instance, H^gare 4.shows edge pixels 
^2(6,2) and Ei(6.3) belonging to a second and first edge feature re^ectively), ften it is 
necessary to discriminate between fliem so that the wrong intensity value is not 
averaged tt^etherviilfa the pixd under consideration. Thus, it is desuable to average 
together neighboring (adjacent) pixels that are both edge pixels and have "smular" 
intensity values with fiie pixel considered for noise removal. Intensify values are 
determined to be amilar to hy comparing the absolute Wue diBeraice b^we^i the 
pixel under consideration for noise rtfmoval and «ljas^ paxel with the average of all 
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such absolute value differences between the pixel under consideration for noise removal. 
If the difference is less than tibe average, fliea the adjacent pixel is considered to be of 
"similar" intensity with the pixel under consideration for noise removal. 

Figure 6 illustrates a flow diagram to perform such nearest neighbor averaging 
of similar intensity pixels. First, according to step 605, the AVD (absolute value 
difference) or distance between the pfocel x(ij) and each adjacent pixel is determined. 
For instance, the north neighboring adjacent pfatel x(i-lo) has a distance <!!^0KrH= 
x(i-l j^. Likewise, the AVDs of each adjacent pixel to the pixel x{ij) is determined. 
The AVD is essentially a measure of correlation between flic iMxel under considemUon 
for noise ranoval and the adjacent pixel which is being cons»d^ed fi» induaon into flie 
averaging set (see step 620). Those a^acent pixels that are well conclated linearly with" 
the pixel x(i j) will be averi^ed together wifli fbs value j) to attain flw iKnse removed 
output pfacelx'Oj). The AVDs thus, are: 

ANQRnf' |x(iJ>x(i-l Asounf |x(ij)-x(i+l j)| 
AwE!?r= Ku>x(ij-l)l, Aeasi- |x(y)-x(i j+l)| 
ANORriffiAsi- !x(ij)-xO-l j-l)|. A50TOWEST= Ix(ij>x(i+g+i)| 
Anortowest= |x(ij)-x(i-l J+1)|, AsourHHAsf = KyHG+l J-l)|. 
Once all (^|ht AVDs (one for each adj acent iHxd in the ei^t cardinal diiectiaQs) 
is detemtined, then flie AVD aveiage value is detannined (step 610). theAVDaverag? 
value is simply the sum of the individual AVDs divided by dgjit 

Once tf» AVD average value is determined, flie step is cfaedc whether a 
first adjacoit pixel is an edge pfacel or not (step 612). If the [Hxel is not an edge pixel, 
fli^ it cannot be suited for the averaging method and is excluded itom the inxel 
averi^g set (step 640). If tiie adjacent pbcel is an edge pixel, then another comparison 
for reladve correlation is performed to Ghedc whether the adyacent i»xb1 behMigs to tiie 
san» edged feature as Has pixel under consld^tation, or belongs to different edge feature 
(see Figure 4, showing jMxels Ej, E2,'E3and belonging to four different edge features 
of varying intensity. The test for cwxeMon ( step 6 1 5) is performed i^n discovering 
that the adjacent pixel is an edge pixel. According to step 615, the adjacent pixel's 
AVD is compared the average AVD value detemuned at siep 610. If the 
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adjacent pixel is not correlated, i,e., ite value is not less tiian the average AVD, then the 
pixel is excluded from the averaging set (step 640). 

If the adjacent pixel is an edge pixel and its AVD is less than the average AVD, 
then the pixel is included into the averaging set which is utilized for the noise removal 
technique (step 620). As part of the averagmg set, the first step is to add the intensity 
value of that adjacent pixel to a variable (register) ACCUM which essentially keeps a . 
rtuuung sum of each averaging set pixel (step 625). The ACCUM variable or register 
wiU facilitate the averaging determination performed at step 670). Since it is unknown 
at the beginnuig of the noise removal by averaging technique for edge pixels how many 
adjacent pixels are within the avera^g set, a second variable (register in hardware) 
NUMCOUNT is used to track the number of pixels within the averaging set Eveiy 
time a new adjacent pixel is mcluded in fee avwaging set, KUMCOXJNT is inciemeated 
by one, starting at an initial value of zero. 

If any otiier adjacent pixels are remaining to be considered for averaging, then the 
next adjacent pixel is fetched (step 645) and the steps 612 through 650 repeat. When a 
pixel is excluded from the averagmg set, the seune check for other remammg pixels (step 
650) is performed. When there are no more adjacent pixels left to be considered for the 
averagmg set,, then finally, the value of the pixel xO j) under consideration for noise 
removal is added to ACCUM (step 655). This fecilitates the averting in of the original 
pixel value with the averaging set so that the original pixel value has equal 
representation therein; In addition, NUMCOUNT is incremented by one (step 665) 
so as to account for the extra pixel x(ij) included in flie averaging set. According to 
step 670, the average of the pixels in flie averaging set is determmed by dividmg the 
contents of ACCUM by NUMCOUNT, The resultant average is the noise removed 
pixel x'(ij). This \^ue x'(ij) is stored into an array or table so that the original value 
x(i j) is not overwritten and can be used in fiiidier noise removal computation for other 
pixels (step 675). When the noise removed pixel value x'^j) is detetmiiKd, the variables 
(registers) ACCUM and NUMCOUNT are cleared (set to zero to feciatate the 
processing of 3ie next edge pixel x. When Ihe iu>i8e is iranoved tarn the ed^ pxel 
X, another pixel classified as an edge pixel may be febshid (step 685) and the steps 605- 
680) repeated for that new pixel. 
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By averaging together adjacent edge pixels of similar intensity with the origmal 
pixel x(ij), the edge feature is maintained and may even be enhanced due to the 
smootiiing and raiifonnity across the edge as a result of averaging. Treating the aige 
pixels differently from non-edge pixels avoids the traditional problems of noise removal 
edge decimation and Weeding in color after interpolation is performed. The averaging 
technique may also be implem^ited in a fest hardware architecture as desired in 
portable digital camera, and can apply direcdy to the CPA domain (Bayer pattern ROB 
color space prior to color interpolation). 

Figure 7 is a system diagram of one embodhnent of the invention. 

Figure 7 illustrates a computer system 71 0, which may be any general or 
special purpose determining or data processing machine such as a PC (personal 
computer), coupled to a camera 730. Camera 730 may be a digital camwa, digita] video 
camera, or any image capture device or imaging system, and is utilized to eaptaic a 
sensor image of an object or scene 740. Essentially, captured images ate oompressed 
and processed by an image processing circuit 732 so ibatibsy can be cffidently stored 
in an image memory unit 734, which may be a RAM or other storage device sodi as a 
feed disk, miniature card, ^ In most <%{talcani«raB, raw images obtained from fl» 
image c^ture mechanism such as a sensor are stored first on camm 730 and . 
downloaded later to be output, displi^ed, or manipulated. This allows the camera 730 
to capture Ifae next object qmdbdbrwi^utaddttioDal One convenient raw image 
representation is (he S-Ut Bayer pattern CPA described above. 

In one embodimoiti an image is captured by cameiB 730 as a row 8-bit Bayor 
pattern CPA and then compressed into some odier data w encoded format Noise . 
remo^ according to one embodiment of the invrattion operates as follows. First, the 
compressed image is downloaded from camera 730 tbrous^I/O port 717 to adii^ 718 
or other device on tiie computer ^srstem 710. The compressed image is decompressed 
on computer syston 710. Th^ the computer system performs noise removal on tlie 
recovered CPA ixm^ i»ior to any ookx inbapolation by executing instructions finr 
domg so as described m the various onbo^ents 'of the iiiventicm. 

The methodolo^ desoibed in &e various eqibodimeats of the invention may be 
executed uang a processor 712 such as. the Pentium™ (a product of btel Corporatioa) 
and a m«nory 711, such as RAM, wldcb is used to store/load instruction, addresses 
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and result data. The qjplicatiott used to perfonn noise removal on the CFA image niay 
be an ececutable module compiled fiom source writtett in a language such as C++. The 
instructions of that executable raodtile, which correspond with instructions that aid in 
detecting edges and applyiog a first noise removal technique for edge pixels and a 
second noise rranoval technique for non-edge pixels may be stored to a disk 71 8 or 
memory 71 1, and thus may be loaded through some computer-readable medium. It 
would be readily apparent to one of ordinary ddU in the art of computer sciences to 
program a determining machine to cany out (he inteipoladon nirthodology deaaribed in 
various embodiments of die invention. 

Computer system 710 has a system bus 713 which facilitates information 
transfer to/fiom the processor 712 and memory 71 1 and also has a bridge 714 which 
couples system bus 713 to an I/O bus 715. VO bus 715 connects various I/O devices 
such as a display adapter 716, disk 718 and an I/O port, such as a serial port. Many 
such combinations of I/O devices, buses and bridges can be utilized with the mvention 
and the combination shown is merely illustrative of one such possible combination. 

When an image, such as an image of object/scene 740, is captured, the image is 
sensed as R, G, and B pkels, in a CFA such as the Bayer pattern. These pixel values 
are sent to the image processing cuxjuit 732. Image processing circuit 732 consists of 
ICs and o&er conqxMients which execute among other fimctions, an image compression 
scheme to reduce the size of the transfo- between camera 730 and computer ^sfem 
71 0. The need for color inteipoMon is most significant v^en the image is to be 
displayed or output and thus does not have to be performed on camera 730. When the 
user or application desires/requests a download of images, the compressed images 
stored in the image memory unit are transferred from image memory unit 734 to the I/O 
Port 717. I/O port 717 uses the bus-bridge hierarchy shown (I/O bus 71 5 to brieve 714 
to system bus 713) to temporarily store the image data into memory 71 1 or, optionaUy, 
to disk 718. The compressed images are decompressed by suitable application 
software (or dedicated hardware) and may utilize processor 712 for doing so. The ' 
decompressed image data will be recovered as a 8-bit Bayer pattern CFA similar Or 
identical to the image data as captured by fte sensor of camera 730. 

The dowidoaded image once decoiqnessed wiU diu5 have inxels in their ra^ 
CFA form. If a noise removed image is desired to be output on moni^ 720, ften noise 
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nsmoval accoFdiog to the various embodiments of ihe invention should be employed 
prior to fiMming Ml color pixels through color interpolation should be employed. The 
downloaded and decompressed image is analyzed pixel-by-pixel and by applying the 
techniques discussed above, each pixel is transformed into a noise removed pixel. 
Preferably, each noise removed pixel is stored in memory 71 1 or disk 718 into an array 
so that the original (recovered) CFA is maintained for the noise removal of other pixels.- 
Once the noise removal is performed throughout the CFA, then the noise removed array 
may be used to the color interpolation process, and if desired, tbs original (recovered) 
CFA discarded. Noise removed and then color interpolated rendered image 750 will be 
rendered with better detail, sharper edges and less noise, given adequate display devices 
(monitor 720 and adapter 716) and will thus, more closely resemble the original 
object/scene 740. The noise removed and color interpolated image may be stcaed (ntto 
disk 718, memory 71 1 and/or output directly to monitor 720 after befaig rendered 
through display adapter 716. 

The compxiter program code for noise removal msy be packaged onto a 
removable computer-readable medium such as iQoppy disk or CD-ROM and may 
accompany ^ftwaie poforms the deconipression of images downloaded from 
camera 730. As with any other software, it can be downloaded/distributed ova a 
network on communications system, or can be avcdlable in firmware. These 
downloaded iniages can be noise iranoved according to tiidr edge features so tte^ 
images will more accurately visually present the object/scene 740 to ffae user. Noise 
removal mxa^ag to another ^bodiment of Has inventicm mi^ also be implemented in 
hardware on the camera 730 itself. 

In an alternative embodim<sat of the invention, ftts image processing drcmt 732 
can be adapted to mclude dnmtry far pecfcHmii^ noise removal onrcamera, such that 
tfie stored linage in iin^niemoiy unit 734 is a iioise removed iniage. T1itB,\^en 
downloaded to a computer system 710 or a rendering device, a first tevel of noise will 
be already eliniinated. In such an embodiment, the image may be noise lonoved at any 
stage in ifae image processing. For instance, tiieimise removal tecbnique(s) of ll» 
various embodimeaits of fhe inv^cm, which are ai^licable direcdy in the CFA 
domain, may be employed after or before color resolution companding. In »ich a 
hardware ranbodim^t, v^re speed is desired, each noise rranoved pixel, as it is h^ag 
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detennined, can be vvritten back directly to flie image metnoiy unit 734 or olho^ ston^e 
mechanism wbere the CFA is stored, so that the noise removal process is recursive. 
The methods desoibed above for finding pixel gradients and detennining edges are 
easily implemented in VLSI (Very Large Scale Int^ration) or oOxa intcgraied circuitiy . 
The noise removal teduiiques presented in various mbo(&nents of the invention have 
the advantage of potential direct hardware .iiiq}lementation since unlike traditional noise 
removal techniques, fliey are designed to apply directly to the CFA domain. 

The exemplary ^bodiments described herein are provided merely to illustrate 
the principles of the mvention and should not be construed as limiting the scope of tiie 
invention. Rather, the principles of the invention may be applied to a wide range of 
systems to achieve the advantages desmbed heriein and to achieve other advantages or 
to satisfy other objectives as well. 
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CLAIMS: 

What is claimed is: 

1. A method cwnpiisiiig the stops o£ 

classifying pixels of a capttued image vHait in its Color Filter Anay fbnn as 
eito edge pixels or iion*edge pixels; 

applying a first noise r^iioval tedimqie to ttiose 
pixels; and 

applying a second noise removal lechniqae to tbose classified as edge pixels, said 
noise removal tedmiques appikd while said image is still in its CoIckt Filter Anay fann. 

2. A method according to claim Iwheiein the st^ of classifying includes the 
steps of: 

defining a localization re^on, said localization region a portion of said captured 

imag^ 

detenmning the gradient of every pixel \vi11un the localizati 

for eadi pixel, con^aring its gradi^t value to a given threslK>ld value, if the 

gradient value, is greater than said threshold value, classifymg the pixel as an edge pixel, 

and if not, classifying the pixel as a non-e<^ pixel. 

3 . A mediod according to chim 2 wl^rein said given tboeshold value is a 
localized threshold value determined by analyzmg the gradient values for all pxels widiin 
die locaUzaii<m v^on. 

4. A mettiod acconiybg to claim 2 wtKrem the steps of defining, detenmntng 
and classifying ate repeated until all pixels of said ci^tmed image are classified. 

5 . A method according to claim 1 vlierein said fust noise removal technique 
includes the application a multilevel median hybrid filter. 

6. A m^od acoosding to claim 1 wherein said.second noise lemoval 
technique inchides a linear averagii^ of a($acem {»xek bdongii^ to ifae ^ 
feature, 

7. A method acoxcdii^ to claim 2 wherein the step of determining^ 
gradient includes the steps of : 

applying a first mask to a neigihborhood around each pixel whose gradient is * 
being determined to attain a first direction differential quantity; 

t^lying a second mask to said neighborhood around each pixel whose gradient 
is beang determined to attam a secoiKl direction dSfferential quantify, said second 
directioti, wherein said second direction is perpendicuiar to said first directian; 
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detemuDuig the gradifiitt to be the 
addition of said first ditection differential quantily mi said second dn«ction differential 
quantity. 

- 8. A mettiod acooidiDg to claim 7 wiierdui said ou^nitude is obtained by the 

steps of: 

squaring said first direction ^fferential; 

squaring said second direction differential; . 

summing said squared first and second direction differentials; and 

extracting the square root of said sum, said magnitude set to equal said square 

root. 

9. A mefliod according to claim 7 whercm said first mask is a Sobel maslc 
having coefScients of -1 and +1 in variously said first direction. 

10: A method according to claim 7 wherein said first mask is a Sobel mask 
having coefScients of -1 and +1 variously in said second direction. 

11. A method according to claim 7 vsiierein said first mask is a Prewitt mask 
having coefiBciaits of -1, -2, +1 and +2 variously in said first direction, 

12. A method according to claim 7 wherem said first mask is a Prewitt mask 
having coefBci«its of -1, -2, +1 and +2 variously in said second ditection, 

13. A mefliod according to claim 1 vidierein the steps of classifying and 
^plying are performed independently with respect to each color plane in said CFA. 

14. A method according to claim 13 v<*erein said CFA indudes four color 
planes, namely, a Red color plane, a first Green color pian% a secmi Gieai color plane 
and a Blue color plane. 

15. A method according to cknm 1 wberdn said CPA is arranged in a Bayo: 
pattern, 

16. A method accordhig to claun 6 wherem one edge feature is distii^uished 
fiom a another edge feature by q)plying the steps of: 

determiniiig the absolute value difference (AVD) betweoi the pixel under 
consideration for noise removal and each pixel acyacent to raid pixel undor considetatiion 
noise removal; 

averaging tc^etfier all of said AVDs; 
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compaiing said average of AVDs with the AVD for each adjacent pixel, if said 
AVD for a given adjacent pixel is less than said average of AVDs, then including that 
adjacent pixel as belonging to the edge feature of said pixel under consideration for noise 
rranoval. 

17. A computer-readable medium having stored thereon a plurality of 
instruction sequences, said plurality of instruction sequences including instruction 
sequences which when executed by a pixwessor, cause said processw to perform the 
steps of:- 

removing noise from a jnxel classified as a non-edge pixel accotding to a 
first noise removal technique; and 

removing noise from a {axel cias^fied as an e(%e pixd acccniding to a 
second noise removal technique. 

18. An imaging apparatus comprising: 

an image processing circuit to receive raw image data, said I' mge 
processing unit c(nifiguied to remove noise fiom non-ec^e pixdte according to a first 
noise removal technique and to remove noise, fiom edge inxels according to a second 
noise removal technique. 

1 9. An apparatus accordii^ to claim 16 wbetein said raw image data is 
stored in a Bayer pattern color filter anay. 

20. An apparatus accorditig to daim IS fiirdiercQmprisiiig: 

an imt^ monoiy unit coiqjled to said ioiage pio(»ssng circuit to reccuve 
and store said taw image data as pixels already subjected to said noise removal 
tedmiquf^s. 

21. A system compriang: 

an irai^ge ci^mne device configured to provide raw irnage data in tibfi 
Color Filter Array donuun; and 

a data procesdng unit coupled to said image capture device, said data 
processing unit configured to rranove nmse tarn ssad raw image data vMle sfill is tf» 
CFA domain, said data jsocessing umt configured to apply a first noi^ removal 
technique to pixeb in said raw image data classified as noi^edge pixels and to ^ly a 
second noise removal technique to pixels in said raw ima^ data classified as edge pbcels. 
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22. A system according to claim 1 9 fiirttier compiising a storage medium 
coupled to said data processing unit, said storage medium stming said noise i«moved 

23, A system according to claim 19 fiirther comprising a di^lay device 
coupled to said data processing unit, said display device configured to render said noise 
removed raw image data together with interpolated color cooqmn^ to fonn a fiiU 
color linage upon ou^ fiiom said disp% device. 
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